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guaranteed minimum !^ ' 308 
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Packet Switches by defining 302^ 
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defined by at least a source 
identilier. a destination 
identifier, and a minimum 
data rate although other 
Information can also be 
used. Upon arrival at certain 
netwoiked nodes, packets 
are classified according to an 
SLA by reading the source 
and destination addresses in 

the packet Once classified, I 1 1 /318 

the packets are placed in 
a queue and scheduled Xoa 
transmissi<m. A scheduler 
ensures that packets are 
transmitted at tiie minimum 
defined data rate for the 
SLA. The scheduler may 
use a statistical multiplexing 

method, such as deficit round robin, or deficit golden ratio, which is part of the present invention. The deficit golden laUo method assures 
a mimmum rate to packets for a particular SLA, but minimizes jitter and delay. Further, the present invention implements congestion 
control that does not require nodes to be entirely turned off in congested conditions. 
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METHOD AND APPARATUS FOR INTERCONNECTION OF 
PACKET SWITCHES WITH GUARANTEED BANDWIDTH 



5 CROSS-REFERENCE TO RELATED APPLICATIONS 

ThepresentappUc»tionisrelatedtOi^pUcationSerialNo.09/189,347,fil^ 
November 10, 1998, entitled "Method and Apparatus to Reduce Titter in Packet 
Switdied Networks and to Application Serial No. 09/189,819, ffled November 10, 
1998, entitled "Method and Apparatus to Minimize Congestion in a Packet 
10 Switched Network," both incorporated by reference herein. 

FIELD OF THE INVENTION 
The present invention relates to communication netwoiks, and particularly, 
the present inv«ition relates to providing guaranteed quality of ser^ce in a packet 
IS switched network. 

BACKGROUND OF THE INVENTION 
In conununications technology, there is an ever-increasing demand for high- 
performance networks, and in particular, a demand for high-performance Internet 
20 access. This increased demand has led to the development of improved networks 
capable of handling larger volumes of data with smaller delays. Nonetheless, these 
improved networks each have their own shortcomings. 

Communications networks like the Internet are generally formed with a 
number of transmission links mterconnected with switches. A transmission link is 
25 any medium through which signals are communicated and can be single or rauWple 
twisted pairs, optical fiber, coaxial cable, radio links, or other mediums. A switch 
is a device with one or more inputi)orts and one or more output ports. The switch 
directs bits arriving at an input port to the appropriate output port. Switching in 
communications is accomplished usmg one of two methods: circuit switdiing and 
30 packet switching. 
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Circuit switching has been used since the birth of the modem telephone 
system. Circuit switchmg entails establishing an end-to-end connection between 
two points before any uiformation is conununicated. A telephone connection is one 
example of a circuit-switched connection. In eariy circuit-switched networks, 

5 information could only be communicated over the links if the links were not being 
used by another party and if an end-to-end connection was akeady established. 

To make the circuit-switched network more economical. Time Divirion 
Multiplexmg (TDM) was developed. TDM is a meUiod that enables muWple end- 
to-end connections to share the same link. More specifically, using TDM the 

0 respective data streams are transmitted for a designated time period in a round robin 
feshion over a shared transmisaon link. For instance, refOTing to the illustration of 
Fig. 1, to multiplex N data streams using TDM, the first stream is transmitted for 
T(l) seconds, then the second stream for T(2) seconds, and so on until Uie N-th 
stream is transmitted for T(N) seconds. The cycle then repeats starting again wth 

5 the first stream. Hence, each data stream gets a fiaction of the total transmisaon 
rate for the shared link, i.e., the stream i (i = 1, . . N) gets a fiaction 



m 

[7(lh71[2)+...+TO] 



of the transmission rate of tiie link. So, using TDM, a link witii transmission rate 
R is capable of carrying data streams with rate R(l), R(N) provided tfiat R(l) 
+ ... + R(Nf) is less than R Therefore, a circuit-switched networic with TDM 
provides a guaranteed bandwidth to a connection fi-om the source of the connection 
to its destination. 

Despite the ability to send multiple data streams through a shared link at a 
guaranteed rate for each data stream, when a particular data stream does not need 
its time slot on the link, that fiaction of the link capacity is wasted because other 
data streams are not permitted to use tiiat time slot. Thus, although the circuit- 



wo 00/28705 



PCT/US99/22943 



-3- 

switched networic ofifers the user guaranteed bandwidth available at any time, when 
the user is not utilizing its allocated bandwidth, the link lays idle for the user's time 
slot - unavailable for use by aiqr other party, leading to an ineffident utilization of 
available resources. . 

5 Even though there are meffidencies inherent in drcuit switching, such 

switching has proven satisfectory to carry voice data. Non-voice data, however, 
tends to be more "bursty" in nature. Hence, v/hen a data burst arrives, having to 
wait through TDM cycles for transmission will make the transmission of data 
appear slow — an appearance that might be avoided if all resources were available 
10 for use. 

To avoid the ineflBdendes of drcuit switching, the communications industry 
has made a shift to "packet switching*' for non-voice ^e data transmission. Packet 
switching does not establish a single end-to-rad connection for each data stream. 
Rather, the data stream is divided into "packets,'^ each of which carries data as well 

15 as various identifying information about each packet. Such identifying information 
includes routing information, for instance, source and destination addresses for the 
packet When a packet arrives at a switch, it is stored in a buffer. The switch looks 
at the routing information in the packet and routes the packet to the ^propriate 
output port. The packets are then forwarded one "hop" (or switch) at a time, until 

20 tiiey reach their destination. Each packet in a particular data stream does not 
necessarily follow the same path to its destination as the others. 

In most networking protocols, the networking systein is hierarchically 
divided into subnetworks that are interconnected. The nodes (e.g., personal 
computers) in each subnetwork are addressed by a 'Xayer 2- address (sometimes 

15 called a MAC address). A subnetwork delivers padcets between any two of its 
nodes. These packets are labded witii tiie Layer 2 addresses of the source and 
destination nodes of the subnetwork. 

Subnetworks can also be interconnected to one another with the use of 
spedalized switches. These switches determine how to handle apacket based dtiier 
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on the Layer 2 addresses in the packet or on some other routing information. For 
instance, a switch interconnects a number of subnetworks through its several ports. 
For each of its ports, the switch maintains a list of Layer 2 addresses of the 
computers ofthesubnetwork(s) that is(are) attached to the port. When the switch 
5 receives a packet, the switch checks the port lists and identifies the port that is 
connected to the Layer 2 address for the destination of the packet. The switch tiien 
transmits the packet on tiiat identified port. Other networking protocols 
additionally add special routing information in each packet. That routing 
information may be a connection number or a pair of geographicaUy arranged 

10 addresses, as in the case of the Internet Protocol. 

Since packets fi-om difiFerent data streams are all routed over the same 
transmission lines, a scheduling mefliod by which to place the packets fi-om the 
different streams onto the transmission lines is required. One such scheduling 
mefliod is statistical nmltiplexing (SM). In contrast witii TDM, SM does not 

15 allocate fixed periodic firactions of tiie transmission rate of the link f o the different 
data streams. Instead, the SM schedules packets for transmission depending on the 
actual needs of the different data streams. In its simplest form, illustrated in Fig. 2, 
statistical multiplexing schedules packets on the transmission line in the order of 
their arrival at the switch, i.e., on a first come, first served basis. Although such a 

20 scheduling method does not waste communication resources, it cannot guarantee 
a minimum transmission rate for any particular data stream. The packets simply get 
to their destination when they get there. 

A variation of SM called "deficit round robin" (DRR) can provide better 
transmission rate guarantees and is illustrated in Fig. 3. Under DRR, to multiplex 

25 N data streams, N integers W( 1), . . . W(N), are first chosen as a weight (W) for 
each data stream. Then for th^ first data stream (DSl) up to W(l) bits are 
transmitted. If all of the packets of DSl are transmitted before transmitting W(l) 
bits or if bit W(l) is the last bit of a packet, tiira a second integer D(l) is set to 0. 
But if bit W(l) is not tiie last bit of a packet, tfien additional bits are sent until tiie 
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transmission of the packet is completed, and D(l) is set to equal the number of 
additional bits sent. The procedure is repeated for data streams DS2, . . DSN 
with the corresponding numbers W(2), D(2), . . . W(N), D(N). The cycle then 
repeats starting with DSl, but W(l) is rq)laced with W(l) - D(l). Hence, W(l) - 
5 D(l) bits are transmitted instead of W(l) Qn oiher words, D(l) represents a 
""defidf ' ofDSl). Ifthere are additional bits remaining in a packet, then those D'(l) 
ad(titional bits are sent. The procedure is again repeated for DS2, . . . , DSN in a 
similar manner. When the procedure returns again to DSl, W(l) - D'(l) will 
replace W(l). 

10 Therefore, using DRR, a minimum transmission rate for a data stream DSi 

(wherei= 1, . . N) will generally be the fraction 

[FF(l)+Fr(2)+...+J^A0] 

of the transmission rate for the link. Moreover, if some data streams do not need 
their guaranteed allocation, then the other connections can utilize the unused 
resources and will do so automatically. Many variations of DRR are known in the 

15 art, including Weighted Fair Quaiing (WFQ), Generalized Processor Sharing 
(GPS), and Class Based Queuing (CBQ). 

DespiteDRR's ability to generally provide some minimum transmissioniate, 
the cq)acity for the transmission of data at a particular rate is still not reserved like 
it is in circuit-switched networks. Hence, there is still the possdbility that delivery 

20 of a message will be delayed. Th^efore, most data conmiunication service 
providers, although th^ utilize statistical multiplexing for packet transmisdon, will 
not guarantee any minimum quality of service. **Quality of service" refers herein to 
a minimum bandwidth provided between two points, A and B. 

The advent of Asynchronous Transfer Mode (ATM) seems to combine the 

25 benefits of statistical multipl^dng with the guaranteed quality of sendee of circuit 
switcKng. While ATM sends data in packets, or cells, ATM sets up a ^Virtual 



) 



wo 00/28705 PCT/US99/22943 



-6- 

channer before transmitting any data. All of the frames follow the same route 
along the virtual channel, thereby guaranteeing the order of cell arrival. Further, 
because data is sent in cells, these cells can be statistically multiplexed, and can use 
all available resources efficiently while guarantedng a minimum quality of service. 
5 To allow all incoming data streams access to a shared link, ATM monitors the 
number of packets or cells received from a particular data stream. If the number 
received during a time interval exceeds a threshold de^gnated for the mcoming data 
stream, then packets/cells are discarded. 

Nonetheless, ATM is a complicated protocol and pensive to implement, 
10 requiring complex hardware and network managanent software. Further, in 
addition to setting up and maintaining virtual channels, ATM requires the 
reformatting of received data packets into spedal ATM cdls that include a virtual 
channel identifier and a virtual path identifier unique to ATM. So while ATM has 
been used by telephone sovice provides, because of its expense and compledty it 
15 is not generally used for Internet or other data communications. 

Therefore, a simplified system and method for providing a guaranteed 
quality of service for Internet and other packet-switched data communications is 
desirable. 

20 SUMMARY OF THE INVENTTON 

In order to overcome the deficiencies in communication networks described 
above, a system and method in accordance with the invention provides a ample, yet 
flexible packet-switching ^qrstem that can guarantee quality of service between two 
25 points. In particular, such a system includes a plurality of linked nodes. Packet 
Switches, such as standard routejrs, are coupled to the nodes. 

Service Level Agreements, or SLA's, are defined between pairs of Packet 
Switches and guarantee a minimum quality-of-service (minimum bandwidth) 
between the two packet switches. When a packet arrives at a node from a packet 
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switch, the node inspects certain classification infonnation contained within the 
packet. In one embodiment, such classification information is the source and 
destination identifiers (e.g., addresses) of the packet, while in other embodiments 
classification information additionally includes other information. Using the 
5 classification information, the packet classifies the packet with an SLA. A schedule 
in the node ensures that packets fi-dm each SLA are scheduled for transmission at 
at least the minimum data rate corresponding to the SLA. 

In one embodiment, each SLA has its own queue in the node. In such an 
embodiment, a scheduler schedules the packets for transmission using a statistical 

10 multiplexing method. In one embodiment, such a method is a DRR method. In 
another embodiment, such a method is a modified DKR method referred to herein 
as a ""Deficit Golden Ratio" (DGK) method. A DGR method in accordance with 
the invention, redistributes the order of the packets that would otherwise be ou^ut 
using a DRR method, thereby eliminating jitter and delay often caused by DRR 

IS scheduling. 

In a second embodiment, the SLAs served by a node share a queue. In 
other words, all packets, no matter which SLA they are classified with, are placed 
in the same queue. However, a statistical multiplexing scheduling method referred 
to herein as SLA Early Discard is utilized to ensure the minmium bandwidth of all 

20 SLAs. If the number of packets in the queue for a particular SLA have reached or 
exceeded a threshold value, then no additional packets for that particular SLA are 
, added to the queue. Any newly arriving packets for the SLA are discarded. If the 
packets for the particular SLA have not reached the threshold, then additional 
packets for that SLA can be added to the queue. 

25 In addition to classification and scheduling, nodes in accordance with the 

hivention fiirther implement congestion control. When packets for a particular SLA 
are transmitted at a rate higher than the minimum rate designated for the particular 
SLA, then downstream nodes receiving those packets may become congested. In 
one ^nbodiment, those downstream nodes provide messages to the upstream nodes 
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indicating that the rate of transmission for packets from the particular SLA should 
be reduced. In some embodhnents, the node may turn oflF service for the particular 
SLA while it is congested. However, service to the other SLA's remains turned on 
in the node. 

5 Thus, a system and method in accordance with the uivention can provide 

service to us ws akin to leased lines in a circuit switched system, although the system 
and method described herein is for packet switching. Moreover, such service can 
be provided without reformatting data packets as is done in ATM. In one 
embodfanent, the nodes in accordance wth the invention have ^hraiet interfaces 
1 0 and accept and forward ethemet packets wthout change. Hence, standard packet 
switches "see" a networic in accordance with the invention as an ethemet-type 
network. Finalty, such guaranteed quality-of-service can be provided with minimal 
additional hardware and software unlike other conventional networking metiiods 
such as ATM. 

15 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention is described with respect to particular exemplary 
embodiments thereof and reference will be made to the drawings, which are not 
20 necessarily drawn to scale and in which: 

Fig. 1 is a representational block diagram illustrating time division 
multiplexing; 

Fig. 2isarepresentational block diagram illustrating first-<»me,first-serv^^ 
statistical multiplexing; 

25 Fig. 3 is a representational block diagram illustrating deficit round robin 

scheduling; 

Fig. 4 is a functional block diagram representing a networic in accordance 
with the mvention; 

Fig. 5 is a block diagram illustration of an ethemet frame; 
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Fig. 6 is a &nctional block diagram of a node in accordance with the 
invention; 

Fig. 7 is a representational block diagram illustrating deficit golden ratio 
scheduling in accordance with the invention; 

Fig. 8 is a fiinctional block diagram of a second embodiment of a QoS node 
in accordance with the invention; 

Fig. 9 is a r^resentational block diagram of packets input into a QoS node 
and padcets output from the node usuig SLA early discard m accordance with the 
invention; 

Figs. 10, 11, and 12 illustrate, in block diagram format, congestion control 
in accordance with the invention; and 

Figs. 13(a) and 13(b) are functional block diagrams representing 
reconfiguration of a ring in accordance with the invention. 

DETAILED DESCRIPTTQN 

A system and method in accordance with the invention provides a simplified 
method of packet-switched data communications with a guaranteed minimum 
bandwidth between any two given points. Although a system and method in 
accordance witii the invention is packet-switched, it can provide service much a like 
a leased line in circuit-switched networks. Still, a system in accordance witii the 
invention is not inefficient witii its resources as circuit-switched networks are. 

In order to provide a guaranteed bandwidtii on a padcet-switched data 
communication system, a system and metiiod in accordance witii invention utilizes 
two elements: (1) packet classification and (2) packet scheduling. In some 
embodiments an additional third element, (3) distributed back pressure, is also 
utiBzed. Each of these elements will be described bdow. But first, a genaal 
overview of tfie network to be used in accordance witii the invention is given. 

Fig. 4 shows a generalized fimctional block diagram of a network 100, 
incorporating an embodiment of the invention. As shown in Fig. 4, network 100 
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includes a niunber of nodes 102, which are interconnected to form a ring 104. The 
nodes 102 on the ring 104 are sometimes referred to herein as "ring nodes.** In one 
embodiment, ring 104 is formed with dual rings, which wB be described in more 
detailiater. Coupled to many ofthe ring nodes 102 are other nodes 106, sometimes 
5 referred to herein as "multipledng nodes.** Other than their location on or oS of 
ring 104, ring nodes 102 and multiplemg nodes 104 are substantially the same. In 
Fig. 4, both ^es of nodes are denoted by a circle, "O**, and are generically 
referred to herein as "Quality of Service Nodes" or "QoS Nodes.** 

In one embodiment, the QoS Nodes 1 02, 106 include ethemet interfaces that 

10 operate at IGbps. In other embodiments, however, the nodes may operate at a 
different rate (e.g., 10Mbps, 100 Mbps, 0C12, OC48, OC96, OC192). Moreover, 
some QoS Nodes may operate at one rate while other QoS Nodes op^te at a 
second rate. Hence, the actual rates for either ofthe types of QoS Nodes are not 
intended to be restricted by the invention nor are the interfaces utilized. 

15 Further, the actual links between the QoS Nodes can be formed in any 

manner known to those of skill in the art. For instance, the links interconnecting the 
QoS Nodes can be built from single or multiple twisted wire pairs, optical fibers, or 
coaxial cable. In other embodunents, the Imks can be radio links, or even free space 
infrared links. In addition the protocol used by the links may be based on Gigabit 

20 Ethernet, ATM, WDM, SONET, or other technology. 

Fig. 4 further illustrates the plurality of packet switches 108, denoted with 
a square symbol, The packet switches are similar to those known by persons 
of ordinary skill in the art. For instance, padcet switches can be routes with 
standard ^hemet inter&ces such as those made by Cisco Systems or Nortel 

25 Networics. The packet switches are attached dther directly to ring 104 via a ring 
node 102, or indirectly to the ring 104 via a multiplexing node 106. Each packet 
switch 108 is in turn coupled to a number of LANs or other subnets 1 10, denoted 
by an "X** indicator (only a few subnets are illustrated). 
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As used herein, "packet switches" 108 denote standard packet switches, 
which are known in the art and include devices such as routers. Ring nodes 102 and 
multiplexing nodes 106, although technically packet switches (ance they route 
incommg packets to output ports), are distinguishable from standard packet 
5 switches, and are referred to herein as "Quality of Service Nodes'' or "QoS Nodes " 

Packet Clasfflfication and Scheduling 

In accordance wth the invention, a minimum quality of service (a minimum 
bandwidth) is provided between selected pairs of packet switches, such as from A 

10 to B, B to A, or from A to C. The quality of service for a particular packet switch 
pah- is referred to herein as a Service Level Agreement, or SLA. Each SLA 
includes at least three elements: a source padcet switdi, a destination packet switch, 
and a minimum data rate for data transfix betwera the two padcet switches. For 
instance, referring again to Fig. 4, an SLA may be, for example, a 128-kbps 

IS connection between A and B. 

In one embodiment of the invention, the packets received at each QoS Node 
102, 106 from a packet switch 108 are ethemet fi^es, which spedfy the Layer 2 
address of the source packet switch and the Layer 2 address of the destination 
packet switch. A standard ethemet frame is shown in Fig. 5. A frame mcludes a 

20 preamble field, a start-of-frame (SOF) field, a destination address field, a source 
address field, a length of data field, a data field, a pad fidd, and a checksum fidd. 
These as well as the other fields are well documented and known in the art, 
however, the destination and source address fields are of more importance with 
respect to the present inventioa Thus, m one embodiment of the invention, the 

25 QoS Nodes operate at Layer 2 with ethemet fi:ames. 

Although ethemet fi^es.(packets) are fluently referred to herein, other 
types of packets could be utilized in other embodiments of the invention. Hence, 
ethemet packets are used for puiposes of illustration only, and such illustration is 
not intended to Umit the scope of the invention. 
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First Embodiment 

A first embodiment of a node 102, 106 is shown in Fig. 6. Upon arrival at 
aQoSNode 102, 106, packets(e.g., ethenietfirames)areplacedintoaninputbuflFer 
302. Subsequently, dasrifier304clasfflfieseachpacketinaccordancewi^^ 
5 To do so, classifier 304 reads at least the source and destination identifier of the 
packet to be clasafied, for instance, Layor 2 ethemet firame addresses. The 
dasafier 304 then correlates the pair of identifiers with a corre^onding SLA. In 
one embodimmt, such source and destination identifiers are physical addresses, 
while in other embodiments such source and destination addresses may be 
1 0 composed of other information. 

In addition to source and destination identifiers, other embodiments of the 
invention may utilize other information for packet classification. For instance, the 
type of application (e.g., e-mail) fi-om which the packet originated can additionally 
be used to classify packets wth an SLA Thus, the information used for classifying 
1 5 packets is referred to herein as "classification mformation." 

In one embodiment of the invention, once an SLAhas been identified for the 
packet using the various classification information, the packet is placed into aFIFO- 
type buflFer 306-3 12 that corresponds to the SLA, forming a queue of packets for 
the SLA. Unlike ATM and other protocols^ the packets do not need to be 
20 reformatted or modified in any way. 

Once the packets have been classified according to their SLA, the packets 
are scheduled for transmission by scheduler 3 16 and placed in an appropriate output 
port 317. For purposes of continued discussion, thie scheduling methods used 
herein are discussed with reference only to a smgle output port, although it is to be 
25 understood that a QoS Node could have more than one output port. 

In order to guarantee quality of service fijr each SLA, a scheduler in one 
embodiment of the invention uses the Defidt Round Robin (DRR) scheduling 
method described previously with respect to Fig. 3. With DRR, the rate (R) of the 
link times the weight (W) fi3r a particular data stream divided by the sum of weights 
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is greater than or equal to the bandwidth for the particular data stream. In other 
words, 



RW(i) 



^bandwidth. 



Therefore, usuigtheDRRscheduliiigmethod, aminimumqua%of service for each 
SLA can be guaranteed. 
5 Nonetheless, DRR may introduce delay and/or jitter into the bit stream, 

particularly when there are many queues. In other words, during the time that bits 
for SLA-1 are being sent, bits from the other SLA's must wait Likewise, once 
SLA-l's turn has passed, no bhs will be sent from SLA-1 until bits from all of the 
other participating SLAs have been sent. Such wait times may be unacceptable for 

1 0 some applications. 

Therefore, in accordance witii one embodiment of the invention, a new 
scheduling mefliod is utilized, referred to herein as the Deficit Golden Ratio (DGR) 
method. To reduce jitter, DGR schedules the bits in a similar manner to that done 
for DRR, but rather than imposing wait times, the DGR metiiod interleaves bits 

15 from each of the SLA queues. The distribution is performed utilizing the Golden 
Ratio: 

2 

. More spedficalfy, much as with DRR, the order of transmission for DGR 
is constructed utilizing the wdghts W(k) 0n bits) asagned to each SLA-k, k = 
1,...,K. Anintega'P(k)isthensettoequaltheint^erpartofW(k)/1000. Since 
20 the average padcet loigth in Int^^ Protocol (IP) sqjplications is about 1000 bits, 
P(k) is proximately the avmge nunibar of packets of SLA-k that DRR would 
save in one cycle. In other ranbodiments, P(k) can be found using a denominator 
other than 1000, for instance, if an average padcet length is a different value. 
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The DGR scheduler constructs a packet sequence {packet-1, packet-P} 
for transmission, where P = P(l) + . . . + P(K). The packet sequence is chosen so 
that there are P(k) packets from SLA-k in the sequence. The sequence is selected 
so that the packets from each SLA are distributed in the sequence. The sequence 
S is consmicted as follows. 

First, tile successive dedmal parts of the first P multiples of 0.62 (the golden 
ratio) are calculated by the scheduler. By way of example, if P(l) = 3, P(2) = 1, 
P(3) = 2, then P = 6, and the dedmal parts of the first six multiples of 0.62 are set 
forth in Table 1 below: 
10 TABLE 1 



Multiple 


Decimal Part 


0 


0.00 


1 


0.62 


2 


0.24 


3 


0.86 


4 


0.48 


5 


0.10 



The sdieduler sequmtially groups these multiples into groups corresponding to 
each SLA-Jc, where eadigrouphasP(k) sequoitial multiples. Continuing the above 
example, since P(l) = 3, then SLA-l is designated the first three muWples, SLA-2 
will recdve the next one multiple sinceP(2)= 1, and since P(3)=2, SLA-3 receives 
the last two multiples. The designated decimal parts of the multiples are shown in 
Table 2 below: 



25 
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TABLE2 



SLA 


Designated Dedmal Parts of N 
multiples of 0.62 


SLA-1 


0,0.62,0.24 


SLA-2 


0.86 


SLA-3 


0.48,0.10 



To determine the packet sequence, the decimal parts are placed by the scheduler in 
increasing order, where the designated SLA-k remains correlated to the dedmal 
part. For the example above, such ordering is shown in Table 3: 
10 TABLES 



Decimal Part 


SLA 


0.00 


SLA-1 


0.10 


SLA-3 


0.24 


SLA-1 


0.48 


SLA-3 


0.62 


SLA-1 


0.86 


SLA-2 



The scheduler will output packets from the SLAs according to the re-ordered 
decimal parts. Hence, Table 3 indicates that the packet sequence output by the 
scheduler m our example should be from {1, 3, 1, 3, 1, 2}, where the numbers 
mside the brackets correspond to SLA numbers. 

Using the sequeaice above, the DGR scheduler transmits bits from the SLA 
sequence in a manner similar to DRR. Specifically, the scheduler utilizing a DGR 
method in accordance with the invention first transmits one packet from the first 
SLA in the sequence, in the above example, SLA-l (see Table 3), where this packet 
has Q(l) bits. If Q(l) is greater than or equal to W(l), then the scheduler sets D(l) 
= Q(l) - W(l). SLA- 1 is then dropped form the rest of the current scheduling cyde 
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(i.e., inthe example above, the second and third occurrences of SLA-1 m the Table 
3 sequence would be skipped). Otherwise, the scheduler notes that SLA-1 still has 
W(l) - Q(l) credits. The scheduler then continues this procedure with the other 
SLAs in the DGR sequence, e.g., hi Table 3 above, SLA.3, then SLA-l, and so on, 
sending a packet from each SLA unless the SLA has been dropped from the cycle. 
After it has reached the last SLA iii the sequence, the scheduler starts agam from 
the beginning of the sequence and continues to perform the same procedure until 
all of the SLAs have exhausted all of thdr credits (i.e., until all SLA's have been 
dropped from the cycle). At this point, the cycle ends, and the scheduler starts a 
new cycle with the credits WO) replaced by W(i) - D(i). Upon exhaustion of all 
CTedits by the SLAs, another new cycle starts where the credits allocated to each 
SLA are WO) - D'O), and so on. An example output stream is shown m Fig. 7. 

In this manner, the DGR scheduling method results in smoother traffic and 
smaller delays, but still guarantees the bandwidth reserved for the SLA. 

It should be understood by those of skill in the art that the classifier and 
scheduler described above can be implemented in hardware, software, and/or 
firmware, or any combination thereof 

Second Embodiment 

A second embodiment of a QoS Node 1 02, 1 06 is shown in Fig. 8. Rather 
than having a single queue per SLA, as in Fig. 6, only one queue, an "aggregate 
queue,'* is used for all SLA's in Fig. 8. V/hen packets arrive at a QoS Node from 
various packet switches, the packets are buffered in input buffers 402. The 
classifier 404 maintains a single first-in, first-out quwe in buffer 406 for all the 
SLA's. The packets are placed in the buffer 406 in the order of their arrival at the 
QoS Node. The packets, however, are only placed by dassifier 404 into the queue 
if a scheduler allows such placwnent. Such a scheduler operates in accordance with 
a scheduling method refored to herein as "SLA Early Discard," and represents a 
new form of statistical multiplexing. 
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In accordance with SLA Early Discard, each SLA-k (where k=l,...,K) is 
assigned some integer weight, M(k). When a packet of SLA-k arrives at the QoS 
Node 1 02, 1 06, the packet is accepted into the queue only if the number of packets 
for that particular SLA in the queue is less than M(k). If the number of packets in 
5 the queue for the particular SLA have reached or exceeded M(k), then the packet 
is discarded. For example, if there are three SLAs, ifM(l) = 3, M(2) = 2, and M(3) 
= 2, and ifpadcets arrive at the node in the following order {1,2, 1,3,2, 1, 1,3), 
then the fourth packet from SLA-1 is discarded. Discarded packets can be resent 
from the source or otherwise later recovered in a mannea- known to those of skill in 
10 the art. The resulting packet stream is shown in fig. 9. In other embodunents, the 
number of bits in the aggregate queue for a particular SLA are monitored instead 
of the number of packets. 

Consequently, if all SLA's are busy, then the packet queue contains about 
M(l)+M(2>f . . . +MOc) packets. The link transmits a fraction 

M(k) 

M(l)+M(2)+...+M(^) 

15 of its packets from SLA-k. Therefore, SLA-k is guaranteed the fraction listed 
above of the total link bandwidth. Further, packets are only discarded after 
monitoring the aggregate queue, whereas in ATM, packets are discarded solely on 
the basis of the source they are form. In ATM, regardless of whether other 
resources are required by other data streams, packets are discarded upon reachmg 

20 a threshold for that source during a given time interval. In contrast, the SLA eariy 
discard method of the present invention only discards packets if they exceed a 
percaitage of the aggregate queue. 

For example, if the total memory ^.e., the number of bits the aggr^te 
qu«e can store) for a 1 Gbps link is 2 MB, the delay of the QoS Node (the time the 

25 packet arrives to the time the packet is transmitted) is approximately 16 ms for each 
packet. If 2 MB correspond to approximately 16,000 packets with an average 
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length of 1000 bits per packet, two packets should be allocated for M(k) for 128- 
kbps agreement. That is, if SLA-k has a rate equal to n x 128 kbps, M(k) == 2n if 
the port rate is 1 Gbps. If the port rate is R Gbps then, M(k) = 2nIL For SLA's that 
have a rate equal to large muhiples of 128 kbps, the fluctuations in packet sizes 
5 should not significantly affect the actual guaranteed minhnum bandwidth. 

It should again be understood by those of skiD in the art that the classifier 
and scheduler for a node described above can be implemented in hardware, 
software, and/or firmware, or any combination thereof 

In essence, using dther embodiment as described above, the SLA*s can be 

10 equated to virtual leased lines. In other words, although operating in a packet- 
switched network, minimum bandwidth can be guaranteed between pairs of packet 
switches. Moreover, such guarantees are provided while still utiliang standard 
packet switches and without modifying router software. In feet, one embodiment 
of the invention is compatible with packet switches that include standard Ethernet 

15 interfaces. Yet, the fi^e structure for Ethernet (or other) packets does not need 
to be modified as they would for SONET and ATM, causing the system to appear 
to the packet switch as a shared Ethernet. Nor does a system in accordance with 
the invention require complex hardware and software akin to that required for 
ATM. 

20 

Congestion Control 

As described above, each node will transmit packets for each SLA at a 
minimura rate. Nonetheless, the actual rate provided can actually be faster than the 
minimum guaranteed rate. But if such faster savice is provided, the capadty of 
25 some links in the network may be exceeded. 

Therefore, in addition to the classification and scheduling methods desoibed 
above, in some embodiments tiie QoS Nodes may additional^ implement , a 
congestion control metiiod herdn referred to as Distributed Back Pressure (DBP). 
To avoid link overflow problems, each QoS Node monitors vrfiether any of its 
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SLA*s are congested and maintains a list of all congested SLA's. Such a list is 
represented in Fig. 6 at 3 1 8 and in Fig. 8 at 41 8. The status of an SLA is marked 
as "congested" when the occupancy of the queue corresponding to that SLA 
exceeds some threshold H. The status is reset to "non-congested" when the 

5 occupancy of the queue corresponding to the SLA falls bdow the threshold H. 
Periodically, the QoS Node sends a control message to its ndghboring QoS Nodes, 
including a current list of all congested SLA's. Each QoS Node uses these control 
messages to maintain and update its own list of the congested SLA*s. The 
scheduler (316 in Fig. 6) for each QoS Node then skips (does not schedule any 

0 packets from) every SLA queue marked as congested. Alternatively, the QoS 
Node can sunply reduce the rate of transmission for the SLA, e.g., to the minimum 
guaranteed rates. 

Figure 10 shows three streams A, B, C transmitting with rates 0.2, 0.6, and 
0.2, respectively, and that share an output port of the first QoS Node 702 that has 

5 ah output port rate 1. The SLAs for A, B, and C have minimum rates of 0.2 each. 
However, the scheduler lets stream B be overactive. As a result, the queue for 
SLA-B in the second QoS Node 703 gets saturated: the five SLAs innode 703 each 
get a service rate of 0.2 but the input of the queue for the SLA-B in the second QoS 
Node 703 has rate 0.6. The second QoS Node 703 sends a message to QoS Node 

0 702 indicating that QoS Node 702 should stop U^smitting packets from SLA-B 
or reduce the transmission rate to the minimum guaranteed rate, e.g., 0.6. Hence, 
backpressure is asserted at the source of SLA-B. Only those SLA queues that are 
congested are switched oflF or rate reduced — not the entire QoS Node. In other 
words, the SLAs for A and C will continue to transmit. In contrast, conventional 

5 network flow control techniques actually stops the entire flow from a switch when 
congestion is detected. 

If using an embodimrat of the invention sudi as that as shown in Fig. 8, 
where both the upstream and downstream nodes use aggregated queues, DBF can 
still be utilized. Like the embodiment of Fig. 6, the embodiment of a QoS Node in 
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Fig. 8 provides at least a minimum bandwidth for each SLA. Still, the bandwidth 
provided is permitted to exceed the minimum, which could cause congestion in the 
input queues of other QoS Nodes. When congestion does occur on a QoS Node, 
the congested QoS Node sends a STOP message to the upstream QoS Node 
5 indicatmg that the upstream QoS Node should stop transmitting. After some time, 
the queue at the congested QoS Node empties and the now uncongested QoS Node 
informs the source QoS Node to start transmitting again. Alternatively, in another 
embodiment, rather than sending a STOP ^gnal, the congested QoS Node can 
inform the source QoS Node to serve the aggregate SLA*s at a lower rate, e.g. at 

10 the minimum guaranteed rate. 

For example. Fig. 11 shows DBP when the data streams A, B, and C share 
a single queue in QoS Node 703. AsinRg. 10, the SLA'sfbr A, B, and C have 
minimum rates of 0.2, although the scheduler in QoS Node 702 has allowed SL A-B 
to transmit at 0.6. QoS Node 703 sends a STOP message to node 702, turning oflF 

15 all the streams from A, B, and C. Eventually, QoS Node 703 indicates that the 
streams can resume. As a result, A, B, C together get a service rate in QoS Node 
702 equal to 0.6. This total rate is divided equally between A, B, and C which then 
get 0.2 each. Alternatively, in Fig. 12, when the SLAs A, B, and C share a queue 
in QoS Node 703, QoS Node 703 sends a message to QoS Node 702 to serve at 

20 its guaranteed minimum aggregate rate (0.6) instead of turning oflF. 

In addition, each QoS Node in some embodunents of the invention, can also 
implement an internal flow control. When the occupancy of an output buflfer 3 17 
(Fig. 6) exceeds a high threshold, tfie scheduler 3 16 marks the SLA's tiiat should 
be sent to that buflfer as congested and skips those qu«ies. When the occupanqr 

25 of a queue in an SLA buflfer 306-3 12 reaches a maximum value, tiie QoS Node 
stops transferxing packets from the input buflfers 302 to the SLA queue. When an 
mput buflfer 302 gets full, the etiiemet interfece sends a signal to ^propriate packet 
switches to stop sending packets. 
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The overall effect of this flow control both external and internal to the node 
is to distribute the cumulative packets from an overactive SLA inside the SLA 
queues of the QoS Nodes while keeping the other SLA's flowing, until the packet 
switch itself gets stopped. Accordingly, DBP in accordance with the invention 
5 exerdses a finer congestion control than conventional flow control that shuts off a 
link completely instead of stopping only the overly active data streams. 

Network Bandwidth Management 

In otd&r to manage the bandwidth for the various SLA*s, a pair of 

10 bandwidth managers 1 12 are mcluded in the network as shown in Fig. 4 with a "A" 
symbol. The bandwidth managers determine if a new SLA can be accepted. For 
reliability in one embodiment, two bandwidth managers 1 12 are provided attadied 
to the ring at diflferent ring nodes 102. The bandwidth managers 1 12 monitor the 
current ring configuration and network topology and keep track of the bandwidth 

15 reserved betwera any two pairs of packet switches. Udng this information, the 
bandwidth managers compute both the bandwidth utilized and available on each 
segment of the ring. When a request for a new SLA is made, the bandwidth 
manager uses these computations and determines if the new SLA can be 
accommodated. 

20 The bandwidth managers are, in one embodiment, computers operating 

software to perform the management fimctions described above. A pdr of 
bandwidth managers are provided for reliability. Other embodiments of the 
invention, howev^, could have one or more bandwidth managers. 

25 Network Rec onfiguration 

In some embodiments of the invention, an additional protocol is 
implemented, referred to herein as a 'Tail-Safe Management ProtocoP (FSMP). 
The FSMP protects the networic 100 against fiiilure of the links connecting the QoS 
Nodes as well as &ilure of the QoS Nodes themseh^es. 
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Inoneonbodinieiit, anetwc^ 100 operates withFSMP by includmgasdf- 
healing plqracal layar. Wh«i a feilure occurs in part of the ring, the ring 
reconfigures itsdf to avoid the feUure. FSMP then detects the reconfiguration and 
initializes a network status update. Such self-healing is weB-known in the art such 

5 as in networks uangSOI^T or FDDI protocols. 

In anotho- embodiment, ring 104 is a dual ring, v^ere one ring transmits 
packets in a dodcwise direction and a isecond ring transmits packets in a counter- 
clockwise direction. In case of feilure, the network reconfigures itselffrom a dual 
ring into a smgle logical working ring. Other systems, such as SONET, have also 

0 used dual ring configurations. SONET however, uses its dual rings to 
simuhaneously send two versions of the bit stream: one version clockwise and the 
other counter-clockwise. While this strate©r ensures reUability, it is also very 
wasteful of valuable bandwidth. 

In contrast, an example of FSMP in accordance with the invention is 

5 explained with reference to Figs. 13a and 13b, which illustrates ring nodes 1 . . .M. 
InRg. 13a, each QoS Node on the ring monitors its incoming data streams fi^om the 
dual links. The ring allows transmission in dthra- direction (dockwise or counter- 
clockwise) in an embodiment of the invention for both robustness and to allow the 
shortest path to be chosen fi)r communication between any two QoS Nodes, whidi 

5 hdpsoiable the minimum rates oftheSLAs to be met. Still, unlike SONET, only 
one version of the padcet is sent But i^ for mstance; the incomiiig stream to node 
M stops on the link fiom node M- 1 , as shown in Fig. 13b, then QoS Node M sends 
a message to QoS Node 1 . QoS Node 1 confirms reception of the message to QoS 
Node M. QoS Node 1 then sends a message to QoS Node 2, which confirms 

5 recdpt of the message, and the process continues. When QoS Node M-1 sends a 
message to QoS Node M, QoS Node M will not receive the message and, hence, 
QoS Node M-1 wiU not receive a confirming message. 

Whai QoS Node M does not confirm the message, then faUure of the links 
between QoS Nod M-1 and QoS Node M is confirmed. The QoS Nodes then 
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reconfigure the rings as shown in Fig. 13b, such that a single ring is formed using 
the dual links. The QoS Nodes also initiate a network status update that triggers 
an alarm for the network bandwidth manager to re-run its calculations to determine 
whether the reconfiguration has resulted in any bandwidth reseivatioiis being 
violated (i.e., the question is asked of whether the network can still support the rate 
requir^ents of all the SLAs). 

Time Stamp 

As is known in the art, certain protocols, such as the Transmisaon Control 
Protocol (TCP), penalize connections that fece long delays: the connections with 
short delays are awarded most of the link bandwidth. To avoid such penalties, 
some embodknents of the invention attach a time stamp to a packet as it enters a 
QoS Node firom a packet switch. Once tiie packet has arrived at the QoS Node 
immediately prior to the destination packet switdi, the packet goes through a 
playback buflTer to ensure that the packet leaves T seconds after it entered the first 
QoS Node. In embocfiments implementing tiiis scheme, the QoS Nodes keep tiidr 
clocks synchronized by using a standard clock synchronization method that will be 
known to those of skill in the art. In this manner, all connections are designated a 
minimumdelay, preventing speedy connections firom usuiping excessive bandwidth. 

* ft * 

A system and method has now been described that allows efficient 
operations of a communications network that includes bandwidtii guarantees. Some 
embodiments of the invention fiirther provide delay guarantees as well as regulate 
data flow and provide network reliability. A system in accordance with the 
invention is not only high performance, but also economical, simple, and flexible, 
while remaining compatible witii mudi of the hardware and software already in 
place, particularly hardware and software for routers and ethemet interfeces. In 
essrace, the SLAs for a system and metiiod in accordance with tiie invention can 
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be equated to virtual leased lines. Further, because the network management 
system enables straightforward provisioning and monitoring of SLAs, an 
embodiment of the invention may be particularly useful in a metropolitan area 
network (MAN). 

Itshouldbeunderstoodthattheparticularembodimentsdescribedaboveare 
only illustrative of the principles of the present invention, and various modifications 
could be made by those skilled in the art without departing firom the scope and spirit 
of the invmtion. Thus, the scope of the present invention is limited only by the 
claims that follow. 



10 



WO00/2S705 



PCTAJS99/22943 



-25- 
CLAIMS 

What is claimed is: 

1 . A method of data communications in a network including a plurality of 
5 packet switches and a plurality of quality of service (QoS) nodes, comprising the 

steps of: 

recdvmg, by a QoS node, a plurality of data packets each including 
classification information; 

clasfflfying, by said QoS node, each of sdd data packets with one of a 
10 plurality of service level agre«nents (SLAs) corresponding to said classification 
information for the respective packet; 

without reformatting said data packets, statistically multiplexing, by said 
QoS node, said data packets for transmission in a manner that results in 
transmission of data packets for each SLA at at least a minimum data rate that 
1 5 corresponds to said respective SLA. 

2. The method of claim 1, wherein 

said step of classifying fiirther includes placing each packet into a queue 
corresponding to said respective one of said plurality of SLAs. 

20 

3. The method of claim 2, wherein said step of statistically multiplexing 
includes multiplexing data packets firom each SLA queue in accordance with a 
d^dt round robin (DBR) method. 

25 4. The method of claim 2, wha-dn s£ud step of statistically multiplexing 
includes multiplexing data packets fi-om each SLA queue in accordance with a a 
deficit golden ratio method. 
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5. The method of claim 1, further including the steps of 
assigning an integer value to each SLA; 

said step of statistically multiplexing includes, for each respective data 
packet, if a number of data packets in a data packet transmit queue classified with 
5 the SLA corresponding to the respective data packet is less than the int^er value, 
adding the respective data packet to the end of the queue, but if the number of data 
packets for the respective SLA is greater than or equal to said integer vahie, then 
(fiscarding said data packet. 

10 6. The method ofclaiml, further including the steps of 
assigning an int^er value to each SLA; 

said step of statistically multiplexmg includes, for each respective data 
packet, if a number of data bits m a data packet transmit queue dashed with the 
SLA corresponding to the respective data packet is less than the integer value, 
15 adding the respective data packet to the end of the queue, but if the number of data 
bits for the re^ective SLA is greater than or equal to said integer value, then 
discarding said data packet. 

7. The method of claim 1, further including: 

20 monitoring each SLA for congestion, and if a congested condition is found 

for one of said SLAs, redudng the data rate for transmitting packets for that one 
SLA. 

8. The method of claim 7, wherein said data rate is reduced to smd minimum 
25 data rate. 

9. The method of claim 7, wherein said data rate is reduced to zero. 

10. The method of claim 1, wherem said data packets are ethemet packets. 
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11. A method of data communications in a network including a plurality of 
packet switches and a plurality of quality of service (QoS) nodes, comprising the 
steps of: 

receiving, by a QoS node, a plurality of data packets from at least one 
5 packet switch, wherein each data packet includes clas^cation information; 

clasdfying, by said QoS node, each of said data packets with one of a 
plurality of service level agreements (SLAs) corresponding to said classification 
information for the respective packet; 

addmg, by said QoS node, each packet to a respective one of a plurality of 
10 SLA queues corresponding to the respective SLA with which the data packet is 
classified; 

statistically muhiplering, by said QoS node, the data packets from each 
SLA queue so that for each respective SLA data packets are transmitted at at least 
a minimum data rate that corresponds to said respective SLA 

15 

12. The method of claim 11, wherein said step of statistically multiplexing 
includes multiplexing data packets from each SLA queue in accordance with a 
deficit golden ratio method. 

20 13. The method of claim 11, wherein said step of statistically multiplexing 
includes multiplexing data packets from each SLA qu«ie in accordance with a 
deficit round robin method. 

14. The method ofclaim 12, wherein said classification information mcludes a 
25 source identifier and a destination identifier. 

15. The metiiod of daim 14, wherem said classification information fiirtfier 
includes additional information. 
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1 6. The method of claim 1 1 , wherein the data packets are in an ethemet format. 

17. The method ofclaim 14, further including the step of: 

controlling network congestion by reducing the rate of transmission from 
5 a selected SLA queue in a selected network node. 

18. A method of data communications in a network including a plurality of 
packet switches and a pluralky of quality of sarvice (QoS) nodes, comprising the 
steps of: 

10 receiving, by a QoS node, a plurality of data padcets from at least one 

packet switch, wherdn each data packet includes classification information; 

as^gning a threshold value to each service level agreement (SLA) in a 
plurality of SLAs; 

dassi^uig each of said data packets with one of said SLAs corresponding 
IS to said dassification information for the respective packed 

adding a respective data packet to a transmit queue on a first-come, first- 
served basis if a nurnb^ of data units in said transmit queue classified with the SLA 
for the respective data packet is less than the threshold value, but if the number of 
data units for the respective SLA is greater than or equal to said threshold value, 
20 then discarding said respective data packet; and 

transmitting said data packets from said transmit queue such that for each 
quaUty-of-service agreement said data packets are transmitted at at least a minimum 
data rate that corresponds to said respective quality-of-service agreemmt. 

25 19. The method ofclaim 18, wherdn said clasafication infonnation includes a 
source identifier and a destination identifier. 



20. The method of claun 19, wherein said classification information further 
mcludes additional information. 
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21 The method of daim 1 8, wherein said data units are data packets. 
22. The method of claim 1 8, wherein said data units are data bits. 
5 23 . The method of daim 1 8, wherein the data packets are in an ethemet format. 

24. The method of claim 18, forther including the step of: 

controlling network congestion by reducing the rate of tomsmisaon for an 
SLA in a selected netwoik node. . 

10 

25 . A networic, comprising: 

a plurality of packet switches; ^ 
a plurality of QoS Nodes, inchiding a first set of QoS Nodes linked to form 
a ring and a second set of QoS Nodes, vAiere each QoS Node in said second set is 
15 linked to a respective QoS Node fi-om said first set thereby forming a ring-star 
topology with said first and second sets of QoS Nodes, and wherein each respective 
packet switch is linked to a respective QoS Node so that at least some of said QoS 
Nodes receive data packets fi^om said packet switches; 
wherdn each QoS Node includes: 
20 an input for receiving said data packets; 

a service levd agreement (SLA) packet classifier in communication 
with said input; 

a statistical multiplexing scheduler; and 

a transmit queue m communication with said scheduler and said 
25 classifier; and 

an output in communication with said transmit queue and for 
carrying data packets at a minimum data rate for each of a plurality of 
SLAs. 
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26. The network of claim 25, wherein: 

each QoS node includes a plurality of SLA queues, where each queue 
corresponds to an SLA; and 

said statistical multiplexing scheduler is a defidt golden ratio scheduler in 
S communication with each of said SLA qumies. 

27. The networic of claun 25, wherein: 
said statistical multiplexing sdieduler is an SLA early discard scheduler. 

28. The network of claim 25 vdierein said first set of QoS nodes are linked to 
form a dual ring, and vdierdn each QoS node in said first set is capable of 
cooperating with other QoS nodes to reconfigure the dual ring into a single ring 
upon failure of a link. 

29. A quality of swvice (QoS) node for use in a communication network, 
comprising: 

an input for receivingaplurality of data packets each including classification 
information; 

means for classifying each of said data packets with one of a plurality of 
service level agreements (SLAs) corresponding to said classificationinformationfor 
the respective packet; 

means for statistically multiplexing data packets for transmission fi-om each 
of said SLAs, witiiout reformatting said packets, such tiiat data packets 
corresponding to each SLA are transmitted at at least a minimum data rate 
corresponding to the respective SLA 

30. The QoS node of claim 29, wherein: 
said means for classifying is fiirther for placing each packet into a queue 

corresponding to said respective one of said plurality of SLAs. 



15 



20 



25 
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31. The QoS node of claim 30, wherein: 

said means for statistically multiplexing includes multiplexing data packets 
from each SLA queue in accordance with a deficit round robin (DRR) method. 

5 32. The QoS node ofclaim 30, wherein: 

said means for statistically multiplexing includes multiplexing data packets 
from each SLA queue in accordance with a deficit golden ratio method. 

33. The QoS node ofclaim 29, wha-ein: 

10 said means for statistically multiplexing is further, for each respective data 

packet, if a number of data packets in a data packet transmit queue classified with 
the SLA corresponding to the respective data packet is less than an assigned value, 
for adding the respective data packet to tiie end of the queue, but if the number of 
data packets for the respective SLA is greater than or equal to said assigned value, 

15 then for discarding said data packet. 

34. The QoS node of claim 29, wherdn: 

sdid means for statistically multiplexing is fiirther, for each respective data 
packet, if a number of data bits in a data packet transmit queue classified with the 
20 SLA corresponding to the respective data packet is less than an assigned value, for 
adding the respective data packet to tiie end of tfie queue, but if the number of data 
bits for the respective SLA is greater than or equal to said assigned value, then for 
discarding said data packet. 

25 35. The QoS node of claim 29, further including: 

means for monitoring each SLA for congestion, and ifa congested condition 
is found for one of said SLAs, for reducing the data rate for transmitting packets for 
that one SLA 
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36. The method of claim 35, wherdn said data rate is reduced to said minimum 
data rate. 

37. The method of claim 35, wherein said data rate is reduced to zero. 

38. The method of claim 29, wherein said data packets are ethemet packets. 
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